Fitting Very Flexible Models: Linear Regression With Large Numbers of Parameters

There are many uses for linear fitting; the context here is interpolation and denoising of data, as when you have calibration data want to fit a smooth, flexible function those data. Or de-trend time series or normalize spectrum. In these contexts, investigators often choose polynomial basis, Fourier wavelets, something equally general. They also an order, number basis functions fit, (often) some kind regularization. We discuss how this basis-function fitting done, with ordinary least squares extensions thereof. emphasize that it valuable far more parameters than points, despite folk rules contrary: Suitably regularized models enormous numbers generalize well make good predictions held-out data; over-fitting not (mainly) problem having too parameters. It even possible take limit infinite parameters, at which, if regularization chosen correctly, least-squares becomes mean Gaussian process. recommend cross-validation empirical method model selection (for example, setting form regularization), jackknife resampling estimating uncertainties made by model. give advice building stable computational implementations.